In recent years, "multimedia" which handles audio, video, and other data integratively has been widely utilized. The multimedia is used to transmit information of conventional information media, namely, newspapers, magazines, television, radios, telephones, and so forth, to users. In general, in the multimedia, graphics, sound, images, and so forth, as well as characters, are related to each other simultaneously. It is essential that information of the conventional information media be in a digital format so that it is intended for the multimedia.
Information of each information media is given in terms of amount of digital information. For example, characters require information of 1.about.2 bytes per character, while audio requires information of 64 kbits (telephone quality) per second, and further, moving pictures require information of 100 Mbits or more (current television receiving quality) per second. So, with respect to the information medium of image information, its enormous amount of information cannot be handled in a digital format. For example, although visual telephones have already been put to practical use by means of an ISDN (Integrated Services Digital Network) which accommodates a transmission rate ranging from 64 kbps to 1.5 Mbps, video of a TV or a camera cannot be directly sent over the ISDN.
Accordingly, there is a need for a compression technique for information. In case of visual telephones, a moving picture compression technique according to H.261 standard which is internationally standardized by an ITO-T (International Telecommunication Union-Telecommunication Standardization Sector) is employed. Also, according to an information compression technique conforming to MPEG (Moving Picture Experts Group) 1 standard, audio and video information can be recorded in a normal CD (compact disc) for music.
The MPEG is an international standard for data compression of a moving picture. According to MPEG1, moving picture data is compressed into 1.5 Mbps data, that is, TV signal information is compressed into about 1/100 information. Since a transmission rate of the MPEG1 is restricted to about 1.5 Mbps, according to MPEG2 standardized to meet demands of a higher image quality, the moving picture data is compressed into 2 to 15 Mbps data.
In status quo, MPEG4 is being standardized by a group (ISO/IEC JTC1/SC29/WG11) which has standardized image information for each object on a display screen and required signal processing can be performed, and new capabilities needed for multimedia can be provided.
According to MPEG4, an object shape signal indicating a shape of an object, which is included in an image display signal of each object on a display screen, is coded, and then by referring to the resulting coded object shape signal (hereinafter referred to as a coded shape bit stream), a pixel value signal for color display of the object, which is included in the image display signal is coded.
While values of pixels located inside an object must be coded to display the object, values of pixels located outside the object need not be coded, since the image display signal is processed for each object on the display screen, and therefore information about pixels located outside the object is not necessary.
In MPEG4, therefore, pixel values located inside an object are coded by referring to an object shape, while pixel values outside the object are not, thereby reducing the number of coding bits to-be-coded.
FIGS. 15(a) and 15(b) are diagrams showing coding of a pixel value signal according to MPEG4. Suppose that pixel values of a fish-shaped object Ob displayed on a display screen R are coded herein. Shown in FIG. 15(a) are values of pixels of the object Ob for each macro block Mb, and an inside of the object Ob is represent by dots.
According to MPEG4, pixel values are coded for each unit region (macor block) Mb consisting of 16.times.16 pixels, i.e., for each of blocks into which a display screen R is divided, as in the case of MPEG1 and MPEG2, and therefore values of pixels in a macro block located completely outside the object need not be coded, whereas values of pixels in a macro block including at least a part of the object must be coded. Macro blocks to-be-coded of the object Ob in FIG. 15(a) are macro blocks Mbx represent by oblique lines in FIG. 15(b).
FIGS. 16(a)-16(c) are diagrams also showing the coding of pixel values according to MPEG4. These figures show the coding more specifically. Shown in FIG. 16(a) are pixel values of a brightness signal of pixels of the object Ob for each unit region (subblock) Sb consisting of 8.times.8 pixels, and in FIG. 16(b), an inside of the object is represented by dots.
According to MPEG, motion compensation of pixel values, or coding of coding mode indicating whether to perform intra coding or inter coding, is performed for each macroblock consisting of 16.times.16 pixels. To the brightness signal included in the pixel value signal for color display, DCT (Discrete Cosine Transform), quantization, and coding are performed for each subblock Sb consisting of 8.times.8 pixels, of 4 subblocks into which one macroblock is divided as shown in FIG. 16(a).
While pixel values of a brightness signal of pixels in a subblock Sb located completely outside an object need not be coded, pixel values of a brightness signal of pixels in a subblock Sb including at least a part of the object must be coded. Blocks to-be-coded of the object Ob shown in FIG. 16 (b) are subblocks Sbx represented by oblique lines in FIG. 16(c).
Thus, the numbers of macroblocks Mb and the subblocks Sb in which pixels values are to be coded depends on an object shape, and it is therefore important that the numbers be reduced.
Hereinafter, prior art coding and decoding according to MPEG4 will be described.
FIGS. 12(a) and 12(b) are diagrams showing prior art coding and decoding of an image display signal according to MPEG4, respectively. First, constructions of coding and decoding circuits will be described with reference to these figures.
Referring to FIG. 12(a), there is illustrated a coding circuit referred to herein by a reference numeral 200a, in which the image display signal is coded. The coding circuit 200a comprises a shape coding apparatus 210a for coding an object shape signal Sk indicating a shape of an object, which is included in the image display signal, and a pixel value coding apparatus 220a for coding a pixel value signal Sg with which the object is color-displayed.
The shape coding apparatus 210a comprises a shape coding unit 211a for performing shape coding including arithmetic coding to the object shape signal Sk to generate a coded shape signal Ck, and for performing local decoding (shape decoding) to the coded shape signal Ck to generate a locally decoded shape signal PEk, a shape post-processing unit 212 for performing shape post-processing to the locally decoded shape signal PEk so that an object to be displayed using the signal PEk has a smooth shape, and outputting a shape post-processed signal PFk, and a variable length coding unit 211b for performing variable length coding to the coded shape signal Ck to generate a coded shape bit stream Dk.
The shape coding unit 211a and the variable length coding unit 211b constitute a shape encoder 211. The shape postprocessor 212 corresponds to a noise elimination filter for pixel value coding, for example, a block distortion elimination filter or a mosquito distortion elimination filter.
The coding circuit 200a further comprises a pixel value coding unit 221a for performing DCT and quantization to a pixel value signal Sg included in the display signal on the basis of the shape post-processed signal PFk, to general a coded pixel value signal Cg, and a variable length coding unit 221b for performing variable length coding to the coded pixel value signal Cg to generate a coded pixel value bit stream Dg.
Referring to FIG. 12(b), there is illustrated a decoding circuit referred to herein by a reference numeral 200b, in which the image display signal which has been coded by the coding circuit 200a is decoded. The decoding circuit 210b comprises a shape decoding apparatus 210b for decoding the coded shape bit stream Dk to generate a decoded object shape signal Fk, and a pixel value decoding apparatus 220b for decoding the coded pixel value bit stream Dg on the basis of the decoded object shape signal Fk to generate a decoded pixel value signal Eg for color display of an object.
The shape decoding apparatus 210b comprises a variable length decoding unit 213a for performing variable length decoding to the coded shape bit stream Dk of the object shape signal to generate a variable length decoded shape signal C'k, a shape decoding unit 213b for performing shape coding including arithmetic decoding to the variable length decoded shape signal C'k, to generate a decoded shape signal Ek, and a shape post-processing unit 214 for performing shape post-processing to the decoded shape signal Ek so that an object to be displayed with the signal Ek has a smooth shape, and outputting a shape post-processed output signal as the decoded object shape signal Fk. The variable length decoding unit 213a and the shape decoding unit 213b constitute a shape decoder 213.
The decoding circuit 200b further comprises a variable length decoding unit 222a for performing variable length decoding to the coded pixel value bit stream Dg to generate a variable length decoded pixel value signal C'g, and a pixel value decoding unit 222b for performing inverse quantization and inverse DCT to the variable length decoded pixel value signal C'g to generate a decoded pixel value signal Eg.
Next, operation of the circuits 200a and 200b will be described.
FIG. 13 is a flowchart showing coding of an object shape signal in the prior art coding circuit. In the coding circuit 200a, the object shape signal Sk is coded by the shape coding apparatus 210a for each macro block.
In step S1, the object shape signal Sk is input to the shape coding apparatus 210a externally. In Step S2, the shape coding unit 211a performs shape coding including arithmetic coding to the object shape signal Sk, and outputs the coded shape signal Ck. In Step S3, the variable length coding unit 211b performs variable length coding to the coded shape signal Ck, and outputs the coded shape bit stream Dk. At this time, the shape coding unit 211a outputs the locally coded shape signal PEk in which the coded shape signal Ck has been locally decoded.
Subsequently in Step S4, the shape postprocessor 212 performs shape post-processing to the locally decoded shape signal PEk. Then in Step S5, the processor 212 outputs the shape post-processed signal PFk. After this shape post-processing, an object shape in which the object shape signal Sk has been coded does not show visual distortion in a display screen.
In the coding circuit 200a, the pixel value coding unit 221a performs DCT and quantization to the pixel value signal Sg on the basis of the shape post-processed signal PFk, and the variable length coding unit 221b performs variable length coding to the resulting coded pixel value signal Cg and outputs the coded pixel value bit stream Dg.
FIG. 14 is a flowchart showing decoding of the object shape signal in the prior art decoding circuit.
Initially, in the decoding circuit 200b, the shape decoding apparatus 210b performs decoding to the coded shape bit stream Dk.
In Step S11, the coded shape bit stream Dk is input to the shape decoding apparatus 210b. In Step S12, the variable length decoding unit 213a performs decoding to the bit stream Dk and outputs the variable length decoded shape signal C'k, and further, the shape decoding unit 213bperforms shape decoding to the signal C'k and outputs the decoded shape signal Ek. In Step S13, the shape postprocessor 214 performs shape post-processing to the signal Ek. In Step S14, the shape postprocessor 214 outputs the shape post-processed signal Fk.
Shape post-processing in Step S13 and shape output operation in Step S14 are identical to those in Steps S4 and S5 of the prior art shape coding shown in FIG. 13, respectively.
Further, in the decoding circuit 200b, when the coded pixel value bit stream Dg is input to the pixel value decoding apparatus 220b, the variable length decoding unit 222a performs variable length decoding to the bit stream Dg and outputs the variable length decoded pixel value signal C'g. Then, the pixel value decoding unit 222b performs decoding to the pixel value signal C'g on the basis of the shape post-processed signal Fk from the shape postprocessor 214, and outputs the decoded pixel value signal Eg.
Since the shape post-processed signal PFk output from the shape postprocessor 212 matches the shape post-processed signal Fk output from the shape post processor 214, the pixel value signal can be coded and decoded correctly on the basis of the object shape signal.
In general, the locally decoded shape signal PEk of the object shape signal is referred to when coding the pixel value signal. This is because coding and decoding of the pixel value signal depend on an object shape, and the pixel value signal is decoded on the basis of the decoded object shape signal Ek.
A problem associated with the prior art coding of the image display signal is that the number of bits to-be-coded of the pixel value signal is increased.
FIG. 17(a)-17(c) are diagrams showing prior art shape post-processing. As in the case of FIGS. 15(a) and 15(b) and 16(a)-16(c), an inside of an object is represented by dots, and blocks in which pixel value signals are to-be-coded, are represented by oblique lines.
In description below, a macroblock Mb consisting of 16.times.16 pixels and a subblock Sb consisting of 8.times.8 pixels are both called "block B", since the same thing goes for them.
Shown in FIG. 17(a) is a part of an object shape obtained from the object shape signal to be displayed on a display region consisting of 4blocks B1 .about.B4. Shown in FIG. 17(b) is a part of an object shape obtained from the locally decoded shape signal PEk and the decoded shape signal Ek to be displayed on a display region consisting of 4blocks B1.about.B4. Shown in FIG. 17(c) is a part of an object shape obtained from the shape post-processed signals PFk and Fk to be displayed on a display screen consisting of 4 blocks B1.about.B4.
As can be seen from FIGS. 17(a)-17(c), shape post-processing is important because it reduces visual distortion of the coded object shape signal. However, in some cases, the shape post-processing increases the number of blocks including the object shape.
More specifically, parts of upper right blocks B2 in FIGS. 17(a) and 17(b) which are located outside objects, are, after shape post-processing, located inside an object as shown in FIG. 17(c). Hence, coding pixel value signal of the block H2 becomes necessary. This increases the number of bits in coding a pixel value signal, causing degradation of efficiency in coding.
Hereinafter, a description is given of this shape-post-processing and a mechanism in which a block located outside an object region (hereinafter referred to as a block located outside an object) changes into a block including at least a part of an object region (hereinafter referred to as a block located inside an object) on a display screen, after the shape post-processing with reference to FIG. 18(a)-18(c). In the FIGS. 18(a)-18(c), a shape post-processing target pixel Gx is represented by dots and pixels located inside an object are represented by oblique lines.
In the shape post-processing, it is decided whether the pixel Gx is located inside or outside an object, by using the pixel Gx and pixels Ga.about.Gh located around the pixel Gx. As shown in FIG. 18(a), the pixels Ga, Gb, Gc, Gd, Ge, Gf, Gg, and Gh are located upper left, above, upper right, left, right, lower right, below, and lower right, with respect to the pixel Gx, respectively. Assuming that a coordinate of the pixel Gx is (1, 1), locations of the respective pixels are indicated by coordinates X(0, 0), X(0, 1), X(0, 2), X(1, 0), X(1, 2), X(2, 0), X(2, 1), and X(2, 2).
In addition, in the shape post-processing, it is decided whether the pixel Gx is located inside or outside an object according to decision by majority, in which comparison is made between the number of pixels located inside an object and the number of pixels outside the object, with respect to the 9 pixels Ga.about.Gh.
For instance, in a case where the pixels Ga, Cd, Gf, Gg, and Gh are located inside an object, and the pixels Gx, Gb, Gc, and Ge are located outside the object as shown in FIG. 18(b), since pixels located inside an object are more than pixels outside the object (5 pixels are located inside and 4 pixels are located outside), the decision result is that the pixel Gx is a pixel located inside as shown in FIG. 19 (c).
That is, in a case where there are block boundaries between the pixel Gx and the pixel Gd located left, and between the pixel Gx and the pixel Gg located below, respectively, even if a block including the pixel Gx is a block located outside an object, shape post-processing of the object shape signal results in a block located inside an object.